Fail2Ban filter for hidden files

The problem

A lot of bots are constantly trying to scan your websites, looking for vulnerabilities. A common mistake is having confidential files ( .env, .git/) exposed through the web server.

Here is a part of my nginx access logs.

185.251.19.3 - - [14/Nov/2024:04:06:27 +0530] "GET /database/.env HTTP/1.1" 404 181 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"
45.132.227.143 - - [14/Nov/2024:04:06:28 +0530] "GET /main/.env HTTP/1.1" 404 181 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"
162.19.239.42 - - [14/Nov/2024:04:07:39 +0530] "GET /.env HTTP/1.1" 404 181 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36"
162.19.239.42 - - [14/Nov/2024:04:20:00 +0530] "GET /.env HTTP/1.1" 404 181 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36"
179.43.168.146 - - [14/Nov/2024:04:26:10 +0530] "GET /data/.git/config HTTP/1.1" 404 118 "-" "l9explore/1.2.2"
179.43.168.146 - - [14/Nov/2024:04:26:11 +0530] "GET /.env.example HTTP/1.1" 404 118 "-" "l9explore/1.2.2"
179.43.168.146 - - [14/Nov/2024:04:26:11 +0530] "GET /src/.git/config HTTP/1.1" 404 118 "-" "l9explore/1.2.2"
179.43.168.146 - - [14/Nov/2024:04:47:56 +0530] "GET /assets../.git/config HTTP/1.1" 404 118 "-" "l9explore/1.2.2"
179.43.168.146 - - [14/Nov/2024:04:47:57 +0530] "GET /api/.git/config HTTP/1.1" 404 118 "-" "l9explore/1.2.2"
162.19.239.42 - - [14/Nov/2024:05:04:46 +0530] "GET /.env HTTP/1.1" 404 181 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36"
162.19.239.42 - - [14/Nov/2024:05:18:49 +0530] "GET /.env HTTP/1.1" 404 181 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36"
162.19.239.42 - - [14/Nov/2024:05:49:47 +0530] "GET /.env HTTP/1.1" 404 181 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36"

Now, it is quite clear these aren’t legitimate users. This is where Fail2Ban comes in.

If you are not familiar with Fail2Ban, this is their repository

Solution

Have fail2ban scan the logs to find such requests and ban those IP addresses. This is pretty easy to setup by adding a new filter for fail2ban.

Create a new file in /etc/fail2ban/filter.d/ called nginx-hidden-files.conf

# Fail2Ban filter to match bad requests to nginx

[Definition]
failregex = ^<HOST> - - [.*] "S+ .*/.[^/s]+

datepattern = {^LN-BEG}%%ExY(?P<_sep>[-/.])%%m(?P=_sep)%%d[T ]%%H:%%M:%%S(?:[.,]%%f)?(?:s*%%z)?
              ^[^[]*[({DATE})
              {^LN-BEG}

journalmatch = _SYSTEMD_UNIT=nginx.service + _COMM=nginx

This is assuming having the default log format for nginx. The fail2regex would match any requests trying to access a hidden file/directory.

After creating the file, enable the jail in /etc/fail2ban/jail.conf. Add the following block in the file.

[nginx-hidden-files]
maxretry = 3
enabled = true
port    = http,https
logpath = %(nginx_access_log)

And that is it. Now reload the fail2ban service

sudo fail2ban-client reload

Additional

Since you are using fail2ban, explore the different filters and jail configuration available to secure your server.